Papers with automatic perturbation
Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that supervised models exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. |
| Approach: | They propose a method which automatically generates contrast sets for the visual question answering task by using a semantic input representation. |
| Outcome: | The proposed method computes the answer of perturbed questions, thus reducing annotation cost and enabling thorough evaluation of models’ performance on various semantic aspects. |